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(57) Correction lor color fog is securely achieved 
with limited load of processing. The RGB image data 
are converted into the luminance Y and the chromaticity 
values C 1 . C2 , then a highlight point and a shadow point 
are determined and are connected to define the axis of 
the color solid. Tho color solid is rotated so as that the 
axis becomes parallel to the luminance axis Y and is so 
translated in parallel manner that the pixel of the mini- 
mum luminance coincides with tho original point of the 
color space. 
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Description 

BACKGROUND OF THE INVENTION 
s Field of the Invention 

[0001] The present invention relates to an image processing method, an apparatus and a memory medium therelor. 
Relaled Background Art 

w 

[0002] For adjusting the color ba lance of a photograph ic image there are conventbna lly known two methods, na mely: 

(1) matching the color balance before picture taking; and 

(2) correcting the image after picture taking. 

15 

[0003] As an cxamplo of tho method (1), there is known an color balance adjusting method utilizing a white balanco 
switch for example of a video camera In this method, the white color balance of the taken image is adjusted by pre- 
liminarily taking white paper or the like before starting the main picture taking operation. 

[0004] The method (2) is widely exercised for example in the printing field, but depends on the experience and 
20 intuition of exports. 

[0005] On Ihe other hand, wilh Lhe recent spreading of digital cameras and photo scanners, Ihe digilizalion of Lhe 
photographic image can be easily achieved by the general users. Also the output equipment represented by ink Jet 
printer is progressing toward higher image quality and lower cost, that the general users can easily print the photograph 
al home. 

25 [0006] However, in case of printing a digitized photographic image, the quality of the out putted image is still insuffi- 
cient. 

[0007] For example, for printing the RGB signals from an input equipment by an output equipment represented by 
an ink jot printer through so-called personal computer or the like, there has boon tried color matching between the 
input equipment and the output equipment thorough the CIE XYZ color space by Color Think (trade name) in the Apple 

30 Mcintosh or by CMS in the Microsoft Windows (trade name). It is. however very difficult to execute these adjustments 
in strict manner, because the color reproduction range is naturally different between the input equipment andthe output 
equipment, and thcro is also involved a basic difference in the working principle that the input equipment works on tho 
R, G and B light emissions while lhe oulpul equipment works on lhe C, M, Y and K reflective colors. 
[0008] Even it the color matching is achieved between the input equipment and the output equipment by overcoming 

35 such difficulties, the image satisfactory for the user cannot be obtained merely by strictly reproducing the image from 
Ihe input equipment as a prinl il such original image from Lhe input equipment is not satisfactory. Such situation arises 
in case of an overexposure or an underexposure in the image taking or a dislorled color balance of lhe entire image 
caused by a "color fog 0 phenomenon. 

[0009] For example, in case of auto picture taking by a camera with the AE (automatic exposure) function, there 
40 results a generally dark image of so-called underexposure slate if the background of the object is principally constituted 
by blue sky, but there results an overexposure state if the background Is principally dark, so that the intended object 
is not always recorded in a best state. 

[0010] Also in case of a digital camera, the image is picked up by a CCD, so that a wavelength region not sensitive 
to the human eye is also contained in the imago signal. If such signal is processed as a part of tho RGB signals, the 
« original invisible color becomes visible to distort the color balance. Naturally there is incorporated an-infrared cut-cH 
filtering but such filtering is not necessarily complete, and the color balance cannot be completely corrected within the 
limitation of real-time processing. As a result there is caused a "color fog" phenomenon, distorting the color balanco 
ol lhe entire image. 

[0011] Since similar phenomena may occur in the pholo scanner or lhe Hal bed scanner, Lhe color balance may be 
so distorted in the digitization with such scanner even if the original negative or reversal film is in the optimum state. 

[0012] Therefore, in order to obtain satisfactory out put result it is necessary to correct the input image data themselves 
into the image data ol appropriate exposure with satisfactory color balance, and, for this purpose, there is required a 
simple method not requiring cumbersome operations by the user, with a sufficiently acceptable processing speed. 

ss SUMMARY OF THE INVENTION 

[001 3] In consideration of the foregoing : a concernof the present invention is to enable color balance correction of 
high -definition image data with a simple configuration. 
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[001 4] According to tho present invention thoro is provided an imago processing method o1 dotccting tho luminance 
of a highlight point and a shadow point ot an original image, determining the ctiromaticlty of the highlight point and the 
shadow point from plural pixels of the luminance, and executing a correction process on the original image based on 
tho highlight point, tho shadow point and tho chromaticity: wheroin tho correction process executes matching of tho 
s color solid axis of the original image with an axis indicating the luminance and contrast adjustment on a color component 
indicating the luminance of the original image. 

[001 5] Another concern of the present invention is to enable judgment of the image state by a very simple method 
and optimum correction automatically according to such judgment. 

[0016] According loanotheraspeclol the present invention there is provided an image processing method comprising 
w steps of: 

delecting the color solid axis ol an original image; 

judging the exposure state of the original image from the positional relationship between the color solid axis and 
an axis Indicating the luminance in a color space in which the color solid is represented; and 
i5 setting an image correcting condition according to the result of the judgment. 

[0017] The above-mentioned object can also be attained by an image processing method for effecting an image 
correction process on an original image according to the color distribution thereof, comprising steps ol; 

20 dctocting the color solid axis of tho original imago in a predetermined color spaco; and 

controlling Lhe image correclion process based on the positional relationship of the color solid axis in the color 
space. 

[0018] Still olher advantages of lhe present invention, and the features thereof, will become fully apparent from the 
25 following detailed description, which is to be taken in conjunction with the attached drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0019] 

30 

Fig. 1 is a block diagram of an image processing apparatus; 
Fig. 2 is a block diagram ol a printer driver; 

Fig. 3 is a flow chart showing the control sequence of an image correction process unit: 
Fig. 4 Is a flow chart showing the sequence of preparing a luminance histogram; 
35 Figs. 5A. SB and 5C are schematic views of a color solid indicating the distribution o1 luminance and hue; 

Fig. 6 is a chart showing a non-linear gamma conversion function: 

Figs. 7 A, 7B, 7C and 7D are charts showing features ol overexposure and underexposure on a luminance -salu- 
ratlon plane, wherein Fig. 7A is a chart showing the concept of an overexposure; Fig. 7B is a chart showing the 
concept of an underexposure; Fig. 7C is a chart showing an example overexposure; and Fig. 7D is a chart showing 
40 an example ol underexposure; 

Fig. 8 is a chart showing the manner of determining the color solid axis in a third embodiment; 

Fig. 9 is a flow chart showing the control sequence of image correction; and 

Fig. 1 0 is a flow chart showing the control sequence o1 color fog correction in a second embodiment. 

45 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0020] Now tho present invention will bo clarified in detail by preferred embodiments thereof with rolcrcncc to tho 
attached drawings. 

so [First Embodiment] 

[0021] In the image processing method of the present embodiment, the original RGB image data are converted into 
luminance and chromaticity data indicating the brightness and the hue for each pixel. In this operation the pixels are 
selected in succession, skipping a pixel for which the saturation of the original image data exceeds a predetermined 
55 value, and a luminance histogram is prepared. Then the luminance positions (luminance values) for which the cumu- 
lative numbers from the high and low luminance sides roach respectively predetermined values are determined as a 
white position (highlight point) and a black position (shadow point). In this operation, in order to Improve the accuracy 
of detection of the white and black positions, there may be adopted a process of excluding the high saturation pixels 
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1rom the luminanco histogram. 

[0022] Then there are calculated the average value ot the chromat lefty of the pixel data having the luminance value 
identified as the white position and the average value of the chromaticity of the pixel data having the luminance value 
identified as the black position, and a straight line connecting these points in the color space is identified as tho color 

s solid axis (gray line). The inclination of the color solid axis with respect to the luminance axis Indicates that the color 
solid axis which should originally indicate the change in the luminance only includes a change in the chromaticity, 
namely that there is generated a color tog. The color fog is corrected by executing a conversion process (rotational 
matrix calculation) on all the pixels in such a manner that the color solid axis assumes the normal position in the above- 
mentioned color space. In Ihe following Ihs present embodiment will be explained in Further detail wilh reference to Ihe 

' o attached d rawings . 

[Configuration ol image processing syslern| 

[0023] An example of the system In the present embodiment Is shown in Fig. 1 . For example a printer 105 such as 

is an ink jet printer and a monitor 106 are connected to a host computer 100. The host computer 100 is provided with an 
application softwaro 101 such as a word processor, a tabic calculation, an internet browser etc., an oporating system 
(OS) 102 : a printer driver 1031or preparing print data by processing various drawing commands (image drawing com- 
mand, text drawing command, graphics drawing command etc. ) indicating an output image issued by the application 
101 to the OS 102 : and a monitor driver 104 tor displaying on the monitor 106 by processing various drawing commands 

20 issued by tho application 101. 

[0024] The hosl computer 1 00 is provided, as Ihe hardwares Tor operating Ihese softwares, wilh a cenlral processing 
unit (CPU) 1 08, a hard disk driver (HD) 107, a random access memory (RAW) 109, a read-only memory (ROM) 1 1 0 etc. 
[0025] As an example of the image processing system shown in Fig. 1. there can be conceived a configuration 
employing a personal computer compatible wilh PC-AT of IBM and Windows 95 ol Microsoft as the operating system, 

25 installing a desired application for executing printing and connecting a monitor and a printer. 

[0026] In the host computer 100, based on an image displayed on the monitor, an application 101 prepares output 
image data utilizing text data classified as a text such as a character graphics data classified as graphics and image 
data classified as a natural imago. For printing tho output imago data, the application 101 sends a print output request 
to the OS 102 and sends thereto a group of drawing commands consisting of graphics drawing commands lor the 

30 graphics data and image drawing commands for the image data. In response to the output request from the application, 
the OS 1 02 issues the drawing command group to a printer driver 1 03 corresponding to the output printer. The printer- 
driver 103 prepares print data, printable by the printer 105, by processing tho print request and tho drawing command 
group entered from Ihe OS 102 and transfers such prinl data Lo the printer 105. In case the prinler 105 is a rasler 
printer, the printer driver 103 executes an image correction process in succession on the drawing commands from the 

3S OS, then executes rasterization into a page memory having a capacity of 24 bits for the RGB signals, then, after the 
rasterisation of all the drawing commands, converts Ihe conlenL of the page memory into a dala formal printable by 
the prinler 105, lor example CMYK data, and transfers Ihe converted dala lo Ihe prinler. 
[0027] The process executed in the printer driver 1 03 will be explained with reference to Fig. 2. 
[0028] In the printer driver 103, an image correction process unit 120 executes an image correction process to be 

40 explained later, on the color information of the image drawing commands contained in the drawing command group 
entered from the OS 102. Acorrection process unit 121 for printer rasterizes the drawing commands : utilizing the color 
information subjected to the image correction process, thus generating dot image data on the RGB 24-bit page memory 
Then it executes masking, gamma correction, quantization etc. for each pixel according to the color reproducibility o1 
tho printer, and generates and sends tho CMYK data dependent on tho printer characteristics to tho printer 105. 

45 

[Image Correction Process Unit] 

[0029] In the following there will be explained, wilh reference lo Figs. 3 lo 7D and 9, the process executed in Ihe 
image correction process unit 120 on the original image represented by Ihe image drawing commands. The following 
so process is not executed on the original Image represented by the graphic drawing commands or by the text drawing 
commands. 

[0030] The image correction process unit 120 of the present embodiment executes, as shown in Fig. 3, a histogram 
preparation process (S20) and an image correction process (S30) according to the histogram. The step S20 prepares 
a histogram by a process shown in Fig. 4, and the highlight point and the shadow point of the image are determined 
5£ from thus prepared histogram. 
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[Preparation of Luminance Histogram] 

[0031] Fig. 4 is a flow chart showing the preparation of luminance histogram in the present embodiment, 

[0032] Referring to Fig. 4, a step S1 initiates tho routinolor preparing tho tuminanco histogram of tho original imago. 

s A step S2 determines the ratio of selection of the pixels to be used for preparing the luminance histogram from the 
pixels of the original image. In the present embodiment the luminance histogram is prepared from all the pixels (se- 
lection ratio 1 (or 100%)) In case the Image data to be processed have 350 : 000 pixels. In case the entered image data 
have a number of pixels larger than 350,000 pixels, the pixels are selected (sampled) according to the ratio of the total 
number pixels lo 350.000 pixels. For example il the input image dala are of 3,500,000 pixels, the selection ratio is 

w 3,500/350 = 10, so that the luminance histogram is prepared at a rate of one out of 10 pixels (selection ratio 10 or 
10%). In the present embodiment, the selection ratio is determined from the following formula: 

n = int(total pixel number of object image data/350,000) 

is 

(however, if n < 1 ( n is selected as 1, and n being a positivo number) 
[0033] Then a step S3 resets a counter for managing the line number or sets It at a predetermined Initial value, and 
a step S4 increases the content ol the counter to a line number of the object line. 

[0034] In the present embodiment, tho sampling of tho pixels is exocutod in the unit of a lino. Thoroforo, for a selection 
20 rat to n : the line number Is divided by n, and, if the remainder is 0, the pixels belonging to such line are selected for 
processing (S5-YES). For example : for a selection ratio 10, the line number is divided by 10 and, if the remainder is 
0, the pixels belonging to the line are selected for processing. 

[0035] If the object line is to be skipped and not to be processed, the sequence returns to the step S4. If the object 
line is Lo processed, the sequence proceeds lo a slep S6, and the pixels belonging lo such line are subjecled lo 
25 luminance conversion and chromaticlty conversion in succession. In the present embodiment the luminance conver- 
sion and chromatic ity conversion are executed according to the following equations. However such conversion are not 
limiled lo these equalions but can also be achieved by various equalions: 

^ Y (luminance) = int(0.30R + 0.59Q + 0.1 1 B) (Y boing positive) 

C1 (chromatic ity) = R - Y 

35 

C2 (chromaticity) = B - Y 

[0036] Also in the present embodiment, in order to improve the accuracy ot detection of the white position (highlight 
point) and black position (shadow point), the saturation S of an object pixel is calculated by the following equation and 
40 is compared with a predetermined saturation valuo (Sconst)(S7), and, if tho saturation S is larger, tho information of 
such pixel is not reflected in the luminance histogram: 

Saturation S - sqrt(C1*2 + C2 A 2) 

45 

wherein sqrt(x) is a lunction giving the square root of x, and x'V means x to a power y. 

[0037] Thus : in case S > Sconst, the sequence returns to the step So and the data of the object pixel are not reflected 
in the succeeding process. This is because, as will be explained later, the saturation of the white position is given by 
the average saturation of a group of pixels of high luminance and such saturation indicates an error caused by the 

so color fog, so that the pixels assumed as of originally high saturation had better be eliminated from the calculation ot 
the highlight point. The effect of this process will be explained by a specific example. For example a yellow pixel (R = 
G = 255, 8 = 0) has a luminanco Y -226 and a saturation S = 227 according to the foregoing equations. Thus, this 
pixel has an extremely high luminance and a sufficiently high saturation. Such pixel should be considered as originally 
yellow in most cases, rather than considered as generated by yellow color logging of an originally colorless pixel. If 

ss the pixel of such high luminanco and high saturation is included in tho luminanco histogram, an error is generated in 
the detected white position. Therefore, in the present embodiment, there is determined a certain saturation (Sconst) 
and Ihe pixel of which saturation exceeds such predetermined saturation is not included in the luminance histogram. 
Such method allows to prevent the error caused by the pixels of high saturation In the detected whhe position, thereby 
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improving tho accuracy of the whrto position. 

[0038] After such discrimination in the step S7, the luminance histogram is prepared on the pixels satisfying the 
condition S < Sconst (S8). As the pixel data employed in the present invention have 6 bits (256 density levels) for each 
of RGB data, tho luminance is also converted into 256 levels. Consequently the luminance histogram is obtained by 

5 counting the frequency of pixels in each of 256 luminance levels. 

[0039] Also in the present embodiment, the calculated chromaticity values CI , C2 are retained in the following manner 
since they are used for calculating the average chromaticity of the pixels of each luminance level, in the color fog 
correction to be explained later. In structurally arranged variable with an index from 0 to 255. the frequency, cumulative 
C1 value and cumulative C2 value are sel as Ihree members and the resull ol calculation Tor each pixel is reflected in 

io a member having the index corresponding to the luminance value of the pixel. 

[0040] When the process on the object pixel is completed, there is discriminated whether all the pixels in the object 
line have been processed (S9) t and, il any unprocessed pixel remains in the objecL line, the sequence relurns lo the 
step S6 to repeat the process thereafter. When all the pixels in the object line have been processed, a step S10 
discriminates whether any line remains unprocessed. and : if all the lines have been processed, a step Sll terminates 

is the sequence, but. if any unprocessed line remains, the sequence returns to the step S4 and the above-described 
process is repeated with the object lino shifted to a next lino. 

[0041] The preparation of the luminance histogram with the selection of the pixels of the original image data in the 
above-described manner allows to obtain a histogram with a minimum necessary number of pixels and with consider- 
ation of improvement in the accuracy in the detection of the white and black positions afterwards. 

20 

| Determination of While Position (Highlight Point) and Black Position (Shadow Poinl)| 

[0042] After the preparation of the luminance histogram, it is used for determining the white position (highlight point) 
and the black position (shadow point). In the present embodiment, the black and white positions are delined as points 
25 where the cumulative luminance frequency reaches 1 750 1rom both ends (luminance value 0 and 255) of the luminance 
histogram toward the center thereof. 

[0043] More specifically, the frequency PY of the pixels of a luminance value Y is cumulatively added as PO + P1 
+ ... and the luminance value when tho cumulative frequency reaches 1750 is defined as tho luminance value YSD of 
the black position. Then there is calculated the average chromaticlty of the pixels of the luminance YSD. As explained 
30 in the foregoing, the cumulative values of chromaticity for each luminance value are calculated at the preparation of 
the luminance histogram (cumulative chromaticity values ClNtotal, C2Ntotal for pixels of luminance N), so that the 
average chromaticity values C1SD, C2SD of tho pixels with tho luminance value YSD corresponding to tho black 
position are determined by the following equaLbns: 

35 

ClSD-C1YSDtotal/PYSD 
C2SD - C2YSDtotal'PYSD 

AO 

[0044] The white position is similarly determined by calculating the cumulative frequency P255 -+ P254 and the 
luminance value where the cumulative frequency exceeds 1 750 is defined as the luminance YHL of the white position. 
Then calculated arc the average chromaticity values CI HL, C2HL of the pixols of tho luminance YHL in the following 
manner: 

45 

C1 HL = C1 YHLLotal/PYHL 



SQ C2HL - C2YHLlolal/PYHL 

[0045] The above-described calculations allow to determine tho white position (C1HL, C2HL, YHL) and the black 
position (C1SD. C2SD. YSD) in the (C1 , C2, Y) color space. 

[0046] In Ihe present embodiment, the cumulative Irequency is determined Irom the luminance positions ol the lu- 
ss minance values 0 and 255, but there may be given a certain offset such as determination from the luminance values 
1 and 254. 

[0047] As explained above, a step S20 in Fig. 3 determines the while position (highlight point) and the black position 
(shadow point). 
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[0048] Then a slop S30 in Fig. 3 oxocutos an imago correction process, based on tho whito and black positions 
determined in the step S20. The Image correction process in the present embodiment includes coior fog correction for 
correcting the color fog in the original age, exposure correction for correcting the luminance contrast for optimizing the 
exposure of the original image, and saturation correction for improving the color appearance of tho output image, 
s [0049] Fig. 9 shows the flow of correction process executed by the image correction process unit 120 in the step 
S30 in Fig. 3. At first a rotation matr ix for color fog correction is determined, then color balance (color fog) is corrected 
by the rotation matrix, and the gamma conversion is executed for the luminance signal according to the exposure status 
of the image. These steps will be explained in succession. 

w [Color Fog Correction] 

[0050] Aller Ihe above-described determination o\ Lhe while and black positions of the original image in the (C1 , C2, 
Y) color space, the color fog correction is executed. 

[0051] If the original image is an ideal image without color fog, R = <3 = B on the colorless axis and the calculated 
is chromaticity values for the white and black positions should satisfy ClHL = G2HL = CI SD = C2SD. However in the 
presence of a color fog, tho lino connecting (C1HL, C2HL, YHL) and (C1SD, C2SD, YSD), namely tho color solid axis, 
shows an inclination toward the color of such color fog, in proportion to the level of the color fog. The color fog correction 
can be achieved by a conversion in such a manner that the color solid axis coincides with the Y-axis (luminance axis). 
The conversion may be achieved by rotation and parallel translation of the color solid, or byconversion of the coordinate 
20 system. 

[0052] In lhe presenl embodiment, the color solid of Ihe original image is al lirsl rolaled, taking lhe minimum luminance 
point (lowest point) of the color solid axis as the center of rotation, until the color solid axis becomes parallel to the Y- 
axis. Then the coordinate system is converted in such a manner that the above-mentioned minimum luminance point 
becomes the original poinl ol lhe (C1, C2, Y) space. AFIer such color fog correction, the color solid shown in Fig. SB 

25 becomes as shown in Fig. 5C. The conversion provides a result having the original point at the minimum luminance 
point and the color solid axis coinciding wfth the Y-axis. Fig. 5A shows a color solid of the image data having an ideal 
color distribution without color fog. The above-described conversion brings the color solid after conversion (Fig. 5C) 
■ close to the ideal color solid (Fig. 5A). 
[0053] In the rotation for bringing the color solid axis parallel to Y-axis, the axis and angle of rotation can be easily 

30 determined from the coordinate values of the shadow and highlight points. The method of determining the rotation 
matr ix tor rotating a solid in a three-dimensional space by a desired angle about a desired axis is already known and 
will not be explained in detail. 

[0054] The color log correction can be corrected, as explained in lhe roregoing, by converting each pixel of Lhs 
original image into pixel data (CI , C2 : Y) in the three-dimensional color space represented by chromatlcity values and 
3S luminance, and applying rotation and parallel translation to such pixel data to obtain pixel data (C1\ C2' t Y) for which 
the color solid axis (gray line) connecting the black and while positions coincides wiLh lhe Y-axis and the minimum 
luminance poinl is al the original point of the coordinate system. 

[Adjustment of Contrast and Saturation] 

40 

[0055] In the following there will be explained a method, for further improving the quality of the image by adjustment 
of contrast and saturation, of judging the over/underexposure state of the image in a simple manner and accordingly 
applying gamma correction to the luminance signal. 

[0056] The contrast adjustment is executed by adjusting tho luminance ol tho black position (shadow point) to "0" or 
*s a value close thereto (for example "10") and that of the white position (highlight point) to "255" or a value close thereto 
(for example "245"). 

[0057] In the -following there will bo explained an embodiment of judging tho over- or undoroxposuro state of tho 

image exposure in a simple manner and accordingly applying gamma correction to the image data. 

[0058] Al Hrsl there are determined poinLs T, V in Fig. 5B t showing Lhe minimum distance between the color solid 

so axis to be corrected and the luminance (Y) axis. These points can be easily determined from the geometrical relation. 
[0059] Then the contrast is so adjusted that the luminance component YT 1 of the point T after the color fog correction 
becomes equal to the luminance component YT of the point T As shown in Fig. 6, a point (YT.YT) is taken as an 
inflection point and. if luminance Y* after the color fog correction is smaller than YTV the luminance is corrected to Y" 
by a function given by a straight line a, and, if it is larger than YT, the luminance is corrected to Y" by a function given 

ss by a straight line b. 

[0060] Naturally there may bo oxocutcd a correction indicated by a lino 12 in Fig. 6 t without using T T. 11 tho color 
solid axis Is parallel to the luminance axis, the point pair T T' Is not uniquely determined, and, if T, T are positioned 
outside the luminance range (0. 255) : the point (YT, YT') cannot be taken as the inflection point In such special case, 
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the correction can be mado according to the lino 12. 

[0061] The correction utilizing the closest points T. V of two axes becomes effective particularly on the Image involving 
over- or underexposure. The overexposure is induced by a fact that the exposure for the enter image is influenced by 
a lighter area such as sky. In such situation the input device represented by a digital camera executes suppression of 

fi the color o1 high luminance, reducing the saturation of the high luminance area As will be understood from the color 
solid axis of the image subjected to such suppression of the color o1 high luminance : represented on a saturation- 
luminance two-dimensional plane shown in Fig. 7A, the pixel closest to gray appears in the high luminance area. On 
the other hand, for an underexposure image, the saturation becomes lower in the low luminance area as shown in Fig. 
7B t because of Lhe suppression of the color of low luminance. 

io [0062] In the actual image, the luminance axis of the color solid assumes a form shown in Fig. 7C for an overexposed 
image, on the luminance-saturation plane. On the other hand, an underexposed image assumes a form as shown in 
Fig. 7D. ir lhe aclual color solid is considered lo be displaced from lhe original color solid in lhe ideal slale by Lhe 
influence of image taking conditions or input conditions at the A/D conversion, the points T, T are considered to indicate 
the positions of minimum displacement. In the present invenlbn such displacement is eliminated to obtain appropriate 

J5 gray in a simple manner, thereby correcting the brightness level ol the entire image. 

[0063] It is naturally possible also to use T sololy as means for simply judging the over- or underexposure state of 
the image, prepare look-up tables (LUT) for the under- and overexposure states and execute the gamma adjustment 
of the luminance signal according to the luminance component of thepoint Tor T*. For example the contrast adjustment 
may be made with a cuive having an inflection point at (YT, YT) in Fig. 6. It is thus possible to simply judge, by the 

20 values of T, T, whether the image is over- or under exposed . Stated different, the luminance component of the point 
T 1 , showing lhe lowest saturation on lhe color solid axis, is positioned al lhe high luminance side, (he image shows a 
luminance-saturation relationship as shown in Fig. 7A : and, if the luminance component of the point T' is at the low 
luminance side, the image shows a luminance-saturation relationship as shown in Fig. 7B. Therefore, in an image 
subjected lo Lhe suppression ol lhe color oT high luminance and low luminance, lhe image is considered in an overex- 

25 posure state or in an underexposure state if the luminance component of the point V is respectively at the high lumi- 
nance side or at the low luminance side. 

[0064] On the other hand, the saturation can be easily adjusted by multiplying the color differences Cl. C2 with 
saturation correction coefficients. As an example, tor an increase of the saturation by 20% a saturation correcting 
coefficient of 1 .2 Is adopted, and the saturation can be corrected by: 

30 

cr=i.2xcr 



3S C2 ,, = 1.2XC2 , 

because the saturation is defined by; 
40 Saturation - sqrt(C1 A 2 i C2*2). 

[Inverse Conversion lo RGB Space! 

[0065] The corrections in the present embodiment are completed as explained Above. In this state, each pixel of the 
*s original image is converted from the (R, G, B) color signal data into (CI ", C2", Y") color space data, which have to be 
inscrvc converted into (R', G\ B') color signal data according to the following equations: 

R' = Y° + C1 a 

50 

G' = Y" - (0.3/0.59) X CI" - (0.1 1/0.59) X C2" 
B' = Y" + C2 M 

[0066] In this manner there can be obtained RGB data which are corrected from the original image in the color fog, 
contrast and saturation. 
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[0067] As explained in the foregoing, tho presort embodiment enables sccuro correction of the color fog with limited 
load of processing. 

[0068] Also in the present embodiment, as the sampling condition is selected according to the image data size of 
tho original imago, there can bo maintained a substantially constant relationship between the ontiro "frequency in tho 
5 histogram and the cumulative frequencies for determining the white and black positions, regardless o1 the Input Image. 
Consequently there can be realized satisfactory color fog correction. 

[0069] In addition, by executing the non-linear gamma correction on the entire image so as to maintain the luminance 
at a point where the axis of the color solid of the image to be corrected is closest to the luminance axis, there can be 
achieved conlrasl correction while maintaining the luminance presumably closest lo that ol the original image. 
io [0070] Furthermore It is possible to easily obtain the over- or underexposure state of the image, and to apply a 
gamma correction by selecting a table according to such exposure state. 

[0071] The pixel sampling may also be executed in Lhe unit of a column insLsad ol the unit of a line. 
[Second Embodiment] 

is 

[0072] In tho following thorc will bo explained a second embodiment which takos tho level of correction into consid- 
eration with respect to the first embodiment. 

[0073] As explained in the first embodiment; if the axis of the color solid determined from the image shows an ex- 
cessively large inclination, the correction o1 such axis may result in the deterioration of the image quality Such situation 
20 may arise in case, 1or cxamplo, tho color log is intent ionally caused by tho uso of a color filter, or tho imago is takon 
under Lhe sunsel environment. 

[0074] In such case, such drawback can be avoided by judging that the determined highlight and shadow points are 
erroneous and not executing the correction, or suitably adjusting the rotation angle thereby attenuating the level of 
correction. Such judgment lhal lhe highlight and shadow points are erroneous can be derived Irom lhe direction of Lhe 
25 color solid axis. 

[0075] Since the color of the color tog can be easily |udged from the inclination of the color solid axis, it is possible 
not to execute the color fog correction for an image taken for example with a color filter for obtaining special effect. 
[0076] In such situation, tho anglo between tho direction vector of tho color solid axis and tho luminance axis is used 
for judging that the color fog correction results in a deterioration of the image : and the correction is not executed or Is 

3Q weakened in the level. For example, if the color solid axis is directed toward red and is inclined in excess of a prede- 
termined angle, for example 40°, such image is judged to inherently contain a color fog. The color fog correction with 
a weakened level can bo achieved, for cxamplo, by raising tho color solid axis by a predetermined angle, for oxamplo 
by 20", or lo a predetermined angle, for example an angle of 20* lo Lhe Y-axis. The rolalion matrix for such conversion 
can be easily determined from the rotation axis and the rotation angle. 

35 [0077] The level of the above-mentioned correction can b© manually designated by the user, or set in advance ac- 
cording Lo lhe magniLude of angle and the direction or Lhe color solid axis. In the foregoing description lhe color solid 
of Lhe image showing an inclination of 40° is raised by rolalion by 20', but Lhe level of such correction may be designated 
by the user. The succeeding process is same as in the first embodiment. 

[0078] Also, as the color of the color fog can be easily judged from the direction of the color solid axis, it is possible, 
40 for example, not to execute correction in the direction of the color of filter, for an image taken with a color filter lor 
obtaining a special effect. In such case, there Is designated a color for which the color fog is not corrected, then, a 
rotation axis for conversion is set along the non-corrected color on the C1-C2 plane and the color solid is rotated until 
a plane containing such rotation axis and the color solid axis 1 becomes parallel to the Y-axis. Such method allows to 
correct the color fog only for tho specified color components. 
« [0079] Fig. 10 Is a flow chart showing a sequence Tor Image correction by rotating the color solid to a separately 
designated angle in case the color solid angle is inclined at least by a predetermined angle. At first a step S101 deter- 
mines tho highlight and shadow points, and a step S1 02 discriminates whothor tho color solid axis is inclined at loast 
by a predetermined angle. If it is less than the predetermined angle, steps S104, S105 correct color Tog by such con- 
version as lo bring Lhe color solid axis lo the luminance axis, as in the first embodiment. 
so [0080] On the other hand, if the color solid axis is i ncli ned by at least the predete rmlned angle , a step S 1 03 determines 
a rotation matrix lor rotating the color solid axis by 20" toward the luminance axis, and a step S10S executes color fog 
correction by rotating the color solid with such rotation matrix. In this process s the angle 20° employed in the step S103 
may be altered arbitrarily. 

[0081] It is also possible to set threshold values at two levels, and, if the inclination of the color solid Axis is larger 
5S than a first threshold value (for example 40 ft ), to rotate the color solid axis thereby raising it (for example 20°) to a not 
completely gray levol. If tho inclination is between tho first and second threshold values (for cxamplo 20°), tho conver- 
sion by rotation is not executed, and, It the Inclination Is less than the second threshold value, the convers ion is executed 
in such a manner that the color solid axis coincides with the luminance axis. Such method albws to avoid defective 



9 



EP 0 967 791 A2 



correction lor tho imago having intontional color fog. 

[0082] As explained In the foregoing, ft is possible to set at least two threshold values for correction based on the 
inclination of the axis, namely the direction and angle thereof, of the color solid constituted by the pixels of the image 
data, thereby judging whether the imago correction is to be oxocutcd, not to bo oxecuted or executed with an adjusted 
s level, thus excluding the defects of the special case in a very simple manner. 

[0083] It is also possible not to correct the color fog in a specified color, since the color of the color fog can be easily 
judged from the direction of the sobr solid axis. 

|Third Embodimenl| 

10 

[0084] In the foregoing first to third embodiments, there has been explained the color balance correction based on 
the highlight and shadow points of the image. In Ihe following Ihere will be explained an embodiment of Ihe correction 
on other reference points. 

[0085] At first there is determined the average color difference AE at the highlight and shadow points by the following 
is equation: 

AE = sqrtffCI HL - CI SD) A 2 + (C2HL - C2SD)*2) 

20 [008S] AE should be zero if the color solid axis is parallel to the luminance axis, but becomes larger than 0 II it is 
inclined. Fig. 6 shows this relation on an E (saturation) • Y (luminance) plane. 

[0087] Then several sample luminances are selected between the highlight and the shadow and the average color 
difference is determined as shown in Fig. 8, for the pixels having a luminance Yn in the image and having a saturation 
smaller [nan AEn. AEn may be a predetermined saturation or Lhe average color difference limited lo the direction of 
25 inclination of the color solid axis. 

[0088] Then a line is determined by the least square method on the average color difference of these several points, 
and the process of the lirsL embed imenl is executed, taking such line as the color solid axis. 

[0089] Otherwise there may bo executed non-liner color balance correction by determining a B-splinc curve approx- 
imating the points having the determined color difference and luminance and taking such curve as the luminance axis 
30 thereby bringing the color difference to 7ero. 

[0090] In this mannor tho lino constituting tho luminance axis can be obtain not from the highlight and tho shadow 
only but from the pixels uniformly sample from the entire image. Thus the color fog correction reflecting the character- 
istics of tue entire image can be achieved by converting the image in such a manner that thus obtained line coincides 
with the luminance axis. 

35 [0091 ] The present invention has been explained by preferred embodiments thereof, but the present invention is not 
limited by such embodiment and is subjected Lo various modilicaLions within the scope and spirit of lhe appended claims. 

[Other Embodiment] 

40 [0092] The present invention may be applied to a system consisting of plural equipment (for example host computer, 
interface devices, reader, printer etc.) or an apparatus consisting of a single equipment (such as a copying machine 
or a facsimile apparatus). 

[0093] Also the objects of tho present invention can bo attained in a case where a memory medium storing program 
codes of sequences shown In Figs. 3, 4, 9 and 10 is supplied to the system or apparatus and a computer (or a CPU 

*s or an MP U) of the system or the apparatus reads and executes the program codes stored in the memory medium. 
[0094] In such case the program codes themselves read from the memory medium realize the functions of the afore- 
mentioned embodiments, and the memory medium storing such program codes constitutes the present invention. 
[0095] The memory medium storing such program codes can be, for example, a Doppy disk, a hard disk, an optical 
disk, a magnetooptical disk a CD-ROM, a CD-R, a magnetic tape, a non -volatile memory card or a ROM. 

so [0096] The present invention includes not only the case where the computer executes the read program codes to 
realize Lhe functions ol Lhe aforementioned embodiments buLalso a case where an operating syslem or the like iunc- 
tioning on the computer executes all the actual process or apart thereof under the instruction of the program codes, 
thereby realizing the functions of the aforementioned embodiments. 

[0097] The present invention further includes a case wherein the program codes read from the memory medium are 
ss once stored in a memory provided in a function expansion board of the computer or a function expansion unit connected 
to the computer, and a CPU or the like provided In the function expansion board or the function expansion unit executes 
all the process or a part thereof according to the instructions of such program codes, thereby reading the functions 
of the aforementioned embodiments. 
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Claims 

1. An image processing method of detecting a luminosity of a highlight point and a shadow point of an original image, 
obtaining a huo of said highlight point and said shadow point from plural pixels of said luminosity and executing 
s a correction process on said orig Inal image based on said high II ght point, said shadow point and said hue: wherein 

said corr ection process executes matching of the color solid axis of said original image with an axis indicating 
the luminosity, and contrast adjustment on a color component indicating the luminosity of said original image. 

10 2. An image processing method according to claim 1 : wherein said correction process executes correction on a color 
component indicating the hue of the original image, and adjusting the saturation of the original image. 

3. An image processing method according to claim 1 , further comprising the steps of preparing a histogram based 
on a color component indicating the luminosity of the original image; and detecting said highlight point and said 

is shadow point from said histogram. 

4. Am image processing method according to claim 3, wherein, In the preparation of said histogram, data of high 
saturation are not included in the histogram. 

20 S. An imago processing method according to claim 1 , whorcin said correction process is oxocutod for a photographic 
image and nol executed for a lexl image. 

6. An image processing method comprising the steps of: 

25 detecting a color solid axis of an original image; 

judging an exposure state of the original image from a positional relationship between said color solid axis 
and an axis indicating the luminosity in a color space representing the color solid; and 
setting an imago correcting condition according to a result of said judgment 

30 7. An image processing method according to claim S, wherein said image correcting condition is a condition for 
adjusting the contrast of a component indicating the luminosity of said original image. 

B. An image processing method for ellecling an image correction process on an original image according lo the color 
distribution of the original image comprising steps of: 

35 

delecting a color solid axis ol Lhe original image in a predetermined color space; and 

controlling said image correction process based on a positional relationship of said color solid axis in said 

color space. 

40 9. An image processing apparatus provided with means for detecting the luminosity of a highlight point and a shadow 
point of an original image ; means for obtaining a hue of said highlight point and said shadow point from plural 
pixels ol said luminosity, and means for executing a correction process on said original image based on said 
highlight point, said shadow point and said hue; wherein 

45 said correction process executes matching of the color solid axis of said original image with an axis indicating 

the luminosity and contrast adjustment on a color component indicating the luminosity of said original image. 

10. An image processing apparatus comprising: 

so detection means for detecting a color solid axis of an original image; 

judgment means for judging an exposure state of said original image from a positional relationship between 
said color solid axis and an axis indicating the luminosity in a color space representing said color solid; and 
setting means for setting an image correcting condition corresponding to a result of said Judgment. 

ss 11. An image processing apparatus for executing an image correction process on an original image, corresponding 
to tho color distribution thorcof. comprising: 

detection means for detecting the color solid axis of the original image in a predetermined color space; and 
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control means for controlling said imago correction process based on a positional relationship of said color 
solid axis in said color space. 

1 2. A computer readable memory medium in which a program of an imago processing method is stored, said program 
comprising the steps o1 detecting a luminosity o1 a highlight point and a shadowpoint ot an original image; obtaining 
a hue of said highlight point and said shadow point from plural pixels of said luminosity; and executing a correction 
process on said original image based on said highlight point said shadow point and said hue, wherein said cor- 
rect ion process executes matchin g of the color solid axis of said orig inal image with an axis indicating the I uminosity 
and con Iras I adjustment on a color component indicating the luminosity of said original image. 

13. A computer readable memory medium storing a computer program for realizing: 

detection means for detecting a color solid axis of an original image: 

judgment means for judging an exposure state of said original Image from the positional relationship between 
said color solid axis and an axis indicating the luminosity in a color space representing said color solid; and 
setting means lor sotting an image correcting condition according to a result of said judgment. 

14. A computer readable memory medium in which a computer' program is stored, for realizing: 

detection means for detecting tho color solid axis of an original imago in a predetermined color space; and 
conlrol means lor conlrolling said image correction process based on a positional relationship of said color 
solid axis in said color space, 

wherein said program causes an image correction process to be executed on the original image corresponding 
to the color dislribulion or the original image. 
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